Calibration Accuracy of a Judgmental Process that Predicts the Commercial Success of New Product Ideas

We examine the accuracy of forecasts of the commercial potential of new product ideas by experts at an Inventor’s Assistance Program (IAP). Each idea is evaluated in terms of 37 attributes or cues, which are subjectively rated and intuitively combined by an IAP expert to arrive at a forecast of the idea’s commercialization prospects. Data regarding actual commercialization outcomes for 559 new product ideas were collected to examine the accuracy of the IAP forecasts. The intensive evaluation of each idea conducted by the IAP produces forecasts that accurately rank order the ideas in terms of their probability of commercialization. The focus of the evaluation process on casespecific evidence that distinguishes one idea from another, however, and the corresponding neglect of aggregate considerations such as the base rate (BR) and predictability of commercialization for new product ideas in general, yields forecasts that are systematically miscalibrated in terms of their correspondence to the actual probability of commercialization. Copyright # 2007 John Wiley & Sons, Ltd. key words intuitive judgment; forecasting; calibration; bootstrapping; case-based